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ABSTRACT 

This report summarizes the major activities and 
findings to date of the Career Ladders Research and Evaluation 
Project with respect to the five-year pilot-test currently underway 
in Arizona. Summary descriptions are given of each year's activities 
from 1985 to 1985: (1) development and planning for program 
evaluation; (2) application of research and evaluation 
instrumentation, methodology, reporting procedures and dissemination; 
(3) expansion and dissemination of data base information; (4) the 
emergence of a predictive model for effective career ladders program 
implementation; and (5) advancement in project focus, research design 
and methodology. Descriptive summaries of each year's progress are 
followed by bibliographic listinc* Q f related research reports, 
presentations and publications. Research and evaluation of the 
pilot-test program over a period of time has resulted in the 
following three major findings: (i) successful change and reform can 
be influenced by intervention programs such as career ladders, but it 
is primarily dependent upon well-developed and effective school 
systems; (2) the association between teacher performance and 
competency based on process and developmental evaluation can 
scientifically be related to reliable and valid student academic 
achievement measures; and (3) while the Career Ladders Pilot-Test 
Program does an excellent job of assuring teacher accountability, it 
has been far less successful with respect to assessing the 
corresponding accountability of other key factors of district 
operational effectiveness. (JD) 
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A CHRONOLOGY OF RESEARCH AND EVALUATION OF THE PILOT-TEST CAREER LADDERS 
TEACHER PERFORMANCE AND INCENTIVE PROGRAM 
1985-1990 



Introduction 

Overview 

In 1985 the Center for Excellence in Education at Northern Arizona University responded to the 
opportunity to evaluate the probability of program success of the career ladder pilot-test project initiated by 
the passage of S.B. 1336. In addition to yearly research reports on the current status of the program, the 
objectives of the Arizona Career Ladder Research and Evaluation Project have expanded to include making 
recommendations as to individual districts' readiness to support major change and reform programs. The 
purpose of this report is to trace and summarize the activities of the research and evaluation process related 
to the five-year pilot-test project, from its legislated beginning through the present time, including 
forthcoming activities. Each summary description will be .ollowed by a bibliographic listing of related 
research reports, presentations and publications. 

Figure 1, page 2, presents a diagrammatic overview of the outputs of the research and evaluation 
project. Reading from left to right, this model depicts the process by which data collected by the center is 
systematically analyzed, summarized and formulated into specific policy recommendations to be made to the 
Joint Legislative Committee on Career Ladders (JLCCL) at the end of the five-year pilot-test. This process 
effectively serves as an overall framework for the more specific activities to be described in greater detail and 
in time order below. 

Two major findings have emerged to date which have influenced the direction of resnrch and 
evaluation activities. The first of these is a startling discrepancy in individual districts' readiness to support 
change and reform movements such as career ladders, despite supposedly "uniform" legislative program 
requirements. As a result, evaluation has shifted from overall monitoring of compliance of these 
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guidelines, to include carefiil assessment of the current operating status of support and focus factors for 
change within districts on an individual basis. 

The second conclusion apparent from the first five years of program research is an unrealistic 
expectation which is prevalent with respect to the change process in the social sciences. There has been a 
disturbing tendency for adopters of programs such as Career Ladders to simply "attach" them to existing 
organizational structures and expect both automatic integration and instantaneously successful outcomes. 
What these well-intentioned innovators fail to realize, however, is that new discoveries in the social ana 
behavioral sciences are as evolutionary and developmental in nature as their counterparts in the medical, 
physical and natural sciences. Processes for change and improvement in these fields require a carefully 
planned start-up time, as well as continual monitoring, feedback and adaptation to changing conditions. 
Therefore, a careful understanding of the elements of the change process, as well as a long-run strategic plan 
for evaluating the resulting outcomes, are necessary for a complete and valid evaluation* Research efforts 
during the five-year pilot-test period have incorporated an identification and ^plication of such a model to 
the change and reform process in e< jucation. 

1985 - Development and Planning for Program Evaluation 

Evaluation Design 

Arizona legislation established the Center for Excellence in Education at Northern Arizona 
University as the designated evaluator of the "Career Ladders Pilot-Test Teacher Performance and Incentive 
Program." An evaluation design was formulated and approved, which consisted of an "improvement 
model" requiring districts to follow a yearly formative and cyclical planning, implementation and 
summative evaluation process. Along with evaluation, the Center was to provide monitoring of important 
legislated guidelines. 

Origination of Related Eva luative Components 

In 1984-85, the researcher culminated studies relating to basic human psychological needs and 
organizational functions which result in the greatest worker motivation and production. Several factors 



were identified as being instrumental to successful change efforts. These included the quantity and quality of 
worker competencies, as well as their performance in meeting the objectives of public and private 
organizations. These research results were later incorporated into evaluative components of individual 
districts' career ladder plans. Their primary objective was to assess the districts' ability to provide adequate 
incentives for superior teacher performance. A second aspect of the evaluation process was districts' ability 
to motivate teachers to improve their skills and expand their responsibilities. This research later became a 
part of comparative studies which determine the relationship between a district's "organizational climate" or 
"psychological environment" and career ladders (CL) program success. A second objective was to evaluate 
trend data collected during the pilot-test period in order to determine the direction and extent of impact which 
the CL incentive plan has hud on interpersonal communication within each organization. 

noctoral Dissertations and the Data Base 

An important part of legislative policy intent was to involve university students more actively in 
various phases of the evaluation process. The first doctoral dissertation related to Career Ladders was 
conducted at Northern Arizona University. Recent technological advances with respect to the study 
variables, as well as the complex and broad-based nature of this evaluation process, have resulted in ongoing 
research efforts at all three state universities. The popular effective schools movement, for example, has 
been the focus of a number of current papers, theses, and dissertations. Specific areas being investigated 
include CL program goals of teacher improvement, administrative leadership, and enhanced accountability 
for student achievement. 

Documentatio n 

Packard, R. D. (1985). Arizona career ladders program evf.ln qtion desipn Document on 
evaluation research design presented to the Arizona Joint Legislative Subcommittee on 
Career Ladders, State Capitol, House Wing, Phoenix, Arizona. 

Packard, R. D. (1985, August). Determining administrative e ffectiveness in managing hnm ? n 
resources for development and productivity Paper presented at the Resources 
Management Symposium, Northern Arizona University, Flagstaff, Arizona. 
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Packard, R. D., & Bierlem, L. (1985, December). Developinp and i mplement^ a ran*, r la dd» n 
evaluation process; The Arizona projec t Document distributed by the National 
Association of Secondary School Principals, Washington, D. C, to the participants of 
the Inservice Program for Administrators Conference, Tucson, Arizona. 

1986 - Amplication of Researc h and Rvalnation. Instrumentation , 
Methodology. Reporting Procedures an d Dissemination , 

Related Literature Research 

One doctoral study initiated continuing research and analysis of related literature. It involved an 
extensive review of such key areas as "differentiated staffing" and "incentive plans." In addition, it 
investigated the effects of other, more specific variables related to the many complex programmatic issues 
of concern. These included the following: teacher and adininistrator evaluation, inservice development, 
program input and ownership, change, reform, motivation, communication, leadership, organizational 
management, support systems, curriculum, student achievement, and testing. 

An especially important objective of the literature review was identifying the reasons for the 
success or failure of previous educational reform movements. This focus was necessary, in order for the 
evaluators to identify those :pecific factors which would be most indicative of district "readiness" levels for 
successful program implementation. 

Policy Research 

The initial interest of outside agencies and research centers in the five-year pilot-test eventually led 
to an ongoing association with other social and political entities. Continued input from these agencies has 
extensively influenced research purpose and methodology of the "Arizona Project." Such active 
involvement has helped to determine the potential for legislative bodies to use "policy research" or 
"evaluation research" results in making legislative or policy decisions. 

The Perception Assessment Sr-ate 

The primary career ladders survey instrumentation, entitled the Perception Assessment SmI*. was 
designed in order to elicit information on attitudes and perceptions of organizational functions. This survey 
was developed by a research team composed of members from the three state universities and legislative 
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research analysts, and it was .Initially pilot-tested in two CL districts. The instrumentation included forced- 
choice and open-ended questions related to CL concepts and organizational climate, resulting in both 
quantitative and qualitative deja. (Three years of data have accumulated from the scale which will allow for 
trend analyses.) 

Census Survey. Analysis and Reports of Findinps 

The total population of "Phase I" school districts was surveyed, data were analyzed and initial 
findings were presented to the JLCCL. (Phase n and HI districts were subsequently evaluated as they 
implemented their programs.) More specific and detailed data reports were provided to each school district 
for analysis and improvement of individual areas of weakness. The intent has been for each district to 
evaluate the findings and implement the initial improvement model of refinement and recycling, based on 
research evidence. As a result, change and reform would be based on input from teachers and other 
individuals who have a vested interest in improving themselves and the achievement of their students. 

Documentation 

Packard, R. D. (1986, January). Organizational performance assessment sr.M e Copyright 
Registration Number-TXu 226 869, United States Copyright Office, The Library of 
Congress, Washington, D. C. 

Packard, R. D. (1986, January). The Arizona career ladders research and evaluation pmj «q 
implementation plans, procedures and assignments Paper presented to the first meeting 
of the combined pilot-school districts, Grand Canyon College, Phoenix, Arizona. 

Packard, R. D., & Bierlein, L. (1986, February). Pilot-test and ana lysis of the Qrmuzalhmal 
Performance Assessment Scale . Snowfiake Unified School District, Snowflake, Arizona. 

Packard, R. D., Bierlein, L., Aleamoni, L., & Helmstadter, G. C. (1986, March). Perception 
Assessment Scale. Research instrumentation developed by the Arizona Career Ladders 
Research & Evaluation Team for ihe assessment of the Arizona Career Ladder Teacher 
• Incentive Programs, State Capitol, Senate Wing, Phoenix, Arizona. 

Packard, R. D. (1986, April). Thirteen psychological fac tors in assessment of system 
effectiveness in meeting faculty develop ment nee/t s Paper presented to the National 
Conference on Faculty Evaluation and Development: Lessons Learned, Kansas City, 
Missouri. 

Packard, R. D., Kundin, K., & Bierlein, L. (1986, April). Pilot test review for the Arirnna rareer 
ladders perception assessment scalfl. The Kachina School, Peoria School District, Peoria, 
Arizona. 
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Packard, R. D. (1986, May). A description and projer.Hoj of the effer.ts of the (Wer f» r 
- Excellence in Education as it relat es to career ladder teacher incentive programs on state 
and national educational policy . Paper presented to The Rand Corporation in consortium 
with Rutgers University, The Eagleton Institute of Politics, and Wisconsin Center for 
Education Research, five year study on policy research, Phoenix, Arizona. 

Packard, R. D. (1986, June). Teacher career development and incentive program oi.tlin ft of 
procedures - Paper presented to the Window Rock School District Steering Committee on 
Teacher Development, Center for Excellence in Education, Northern Arizona University 
Flagstaff, Arizona J ' 

Packard, R. D. (1986, July). Implications of career ladder teacher incentive V r^ m Paper 
presented at the 16th Resources Management Workshop, Northern Arizona University 
Flagstaff, Arizona J 

Packard, R. D, Pavlich, P., & Bierlein, L. (1986, Ju,y). The Arizona earner ladder ^ 
evaluation project fact Sheet - Document prepared for general dissemination to education 
and other public sources, University News and Publications, Northern Arizona 
University, Flagstaff, Arizona. 

Packard, R. D. (1986, August). The Arizona career ladder res earch and evaluation project Paper 
presented to the faculty of the Center for ExceUence in Education Workshop, Northern 
Arizona University, Flagstaff, Arizona. 

Packard, R. D. (1986, Fall). Essential elements for assessment of system effective- ss in 
meeting faculty development needs. Issues of Hiphe r Education Jmpj j, Kansas State 
University, Manhattan, Kansas. 

Packard, R. D. (1986, November). A statewide nifot tearher jnr- n tive nrogram- Research and 
development for policy chanfie and reform - Paper presented at the National Council of 
States, Eleventh Annual National Conference, Hyatt Regency Hotel, Nashville 
Tennessee. (ERIC Document Reproduction Service No. ED 275 654.) 

Packard, R. D., & Bierlein, L. (1986, December). Research and evaluation of the Ajfeona H im 
career l a dder teacher inr;ftnfiyft prnpram - Report to the Joint Legislative Committee on 
Career Ladders, 1st Annual Research Report, Arizona House Wing, State Capital, 
Phoenix, Arizona. 

1987 - Expansion and pksernfnati on of Data Rase Information 

Expansion of the Data Bj$g 

A dissertation related to performance-based teacher evaluation expanded the review of related 
literature and analyzed quantitative and qualitative data through basic descriptive, anecdotal and non- 
parametric statistical procedures. As a result of the relationship of success of reform movements and level 
of organizational operations, effective schools research was reviewed and more closely associated with CL 
research findings. In order to satisfy one of the most important concerns of legislative intent, plans were 
initiated which related to the scientific study of accounting for program effects on student achievement 
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Information Di^erninatj on and Reporting of Result^ 

Considerable outside interest in the research results of the career ladders pilot-test program became 
evident during this time. This was a direct result of the focus on relevant issues such as teacher 
development and incentives, as well as their effect upon student achievement Several state, national and 
international presentations related to topics such as the foUowfc g: a) teacher productivity, performance 
evaluation and rewards; b) program designs and structures; c) reform and change in education; and d) teacher 
productivity and instructional leadership roles. 

A major research finding during this time concerned the extreme diversity of individual districts' 
readiness to implement a uniform set of legislative guidelines. This diversity in readiness levels existed 
despite the otherwise carefully matched similarities in CL program goals, designs and structures across 
participating districts. As a result, reporting of project results to the JLCCL became considerably more 
complex. Qurmtitative and qualitative data indicated that factors other than CL plans and goals were, in fact, 
instrumental to potential program success or failure. 

Documentation 

Packard, R. D., & Bierlsin, L. (1987, January). Arizona carer la dder research and » n \ m \^ 
project; Research and development for e ffecri Vft ffdn ra t in n a l c hange and refor m Ann 
c P^^vosity of Michigan, Counseling and Personnel Services Clearing House, 
2108 School of Education (ERIC Document Reproduction Service No. CG 019 609.) 

Bierlein, L. A. (1987, March). The Arizona career ladder pilo t project- Pensions of teacher* in 
partic i p a ting districts toward nfrform anre.ha S ed te acher evaluation p rnf ft ri ti r»i 
Unpublished doctoral dissertation, Northern Arizona University, Flagstaff, Arizona. 

Packard, R. D., & Bierlein, L. (1987, June/July). "Career ladder facts abstract an<« :i,uentive 
programs for teachers: Will it work in Arizona?" Arizona Admin;^ ,- j mtm ] t 16 
(No. 9), 9-10. (ERIC Document Reproduction Service No. ED 287 797.) 

Packard," R. b:*(I987, July). Develonment nf qtoiPJia] Ifiafa Fnrtfrinr ir J ivifllinl 

development of tpartipr for productivity and leadership r n j n jn fr jirnt ; nn Paper 
presented at the World Assembly of tne International Council on Education for Teaching 
Emdhoven, The Netherlands. (ERIC Document Reproduction Service No. ED 283 793.) 

Packard, R D. (1987, October). Research & evaluation: TQR7 pre liminary repnf tfe c^ 
adder teacher incentive and d e velo p m e nt r rft cntm 'Document presented to the Joint 
Legislative Committee on Career Ladders, Arizona State Capitol House Wing, Phoenix 
Arizona. ' 
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Packard, R. D. (1987. October). Executive summary: The 1986-87 nilot test career ladder 
project report- Document presented to the Joint Legislative Committee on Career 
Ladders, Arizona State Capitol, House Wing, Phoenix, Arizona. 

Packard, R. D. & Fargo, S. (1987, November). Diversity of responses a*nn ng ten AnVona pjfrf 
test district career ladder plans* Research document presented to the Joint Legislative 
Committee on Career Ladder Programs, Arizona State Capitol, House Wing, Phoenix, 
Arizona. 

Packard, R. D», Dereshiwsky, M., & Groenendal, L (1987, November). Descriptive & anaiytirat 
results for the 1936-87 career ladder data gygJgS. Research document presented to the Joint 
Legislative Committee on Career Ladder Programs, Arizona State Capitol, House Wing, 
Phoenix, Arizona. (ERIC Document Reproduction Service No. SP 029 861.) 

Packard, R.D. (1987, November). Research & e valuation: 19R7 results for a maior nilot test 
career ladder *escher incentive & develo pment program . Paper presented at the National 
Council of States, Twelfth Annual National Conference, San Diego, California. (ERIC 
Document Reproduction Service No. ED 288 878.) 

Packard, R.D. (J 987, November). North Ce ntral Association evaluation of effective schools 
Ifipon. Document presented to Thunderbird High School, Glendale School District, 
Phoenix, Arizona. 

Packard, R.D. (1987, November). Outline of similarities f c diversities in achieving performance 
targgtS ft Conditions Of nilot test career ladde r programs in the state of Arizona 
Document presented to the Joint Legislative Committee on Career Ladders, Arizona State 
Capitol, House Wing, Phoenix, Arizona. 

Packard, R. D., & Nichols, W. (1987, November). Qualitative analysis ft results for the 1QK7 
data cycle hv career Vtdrier program strenpths ft weaVi^e*. Research document presented 
to the Joint Legislative Committee on Career Ladder Programs, Arizona State Capitol, 
House Wing, Phoenix, Arizona. 

Packard, R. D., & Morrison, L. (1987). Analysis of the initial Arizona career ladder teacher 
incentive programs. Excellence In Teaching 5 (No. 1), 4-6. (ERIC Document 
Reproduction Service No. ED 277 687.) 

Helmstadter,G.C. (1987). An index of teacher effectivene ss which is fair ohf ecrive and m 
Student performance- Paper presented to the Arizona Joint Legislative Committee on 
Career Ladders, Arizona State University, Tempe, Arizona. 



1988 - The Emergence of a Predi ctive Model for Effective 
Career Ladders Program Implen^enffition 



Directional Focus of the Res earch and Evaluation Project 

Research findings during the preceding year had revealed a dichotomy with respect to CL program 
implementation of districts which were otherwise similar in terms of goals and models for designs and 
structures. That is, some of these seemingly equivalent districts were successfully reaching their target 



objectives, while others were failing to make observable progress. As a result, research efforts in 1989 
were focused on identifying and assessing the primary reasons for these dichotomous results. 

The extreme diversities in readiness found in the CL research, combined with private and public 
organizational and effective schools research, identified those operational factors v/ithin the districts which 
influenced performance and production. This ied to the emergence and continual refiriemeEl of an integrated 
model for assessing readiness (see Figure 2, on page 19) of districts for successfully implementing the CL 
program. 

The Career Ladders Pilot-Test Program NetwonV 

The Network is an association of the CL school districts which developed primarily as an 
important support for project research and evaluation. Extensive time and effort has been expended by this 
committee and its research sub-committee in assisting the researchers with evaluation content and collection 
of data. 

Individual School District Research anrf Dissemination 

Several career ladders teacher-leaders, aciministrators and doctoral students within districts initiated 
program research, writing and dissemination of findings in cooperation with the project evaluators at this 
time. A number of specific components of the model have been enhanced through in-depth study of topics 
such as the following: a) production and outcomes in student achievement; b) change theory; c) program 
designs and structures; d) development of district R & D; e) teacher development, leadership, mentoring, 
peer instructional coaching and clinical supervision; and f) cooperation with the development of a predictive 
achievement model. In addition, requests for information from non-career ladders districts throughout the 
state resulted in wider dissemination of CL findings to date. 

Instrumentation Reliability a nd Validity 

The Perception Assessment foalft survey items were evaluated with respect to response consistency 
through calculation of standard reliability indices. A factor analysis was also conducted within each 
subsection in order to identify clusters of questions which best defined each topic area. 
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Documentation 



Packard, R. D. (1988, January). Research questions for the development of rp mpn"""!? ? f ^ 
emerging career ladder pro? ram n^ e ) Document presented to the Career Ladder Pilot 
Network, Mesa Public Schools Administration Center, Mesa, Arizona. 

Packard, R. D. (1988, March). University & nilot career ladde r district collaborative planning 
Document presented to the Joint Legislative Committee on Career Ladders Research Staff 
and the "Network," Mesa Public Schools Administration Center, Mesa, Arizona. 

Dereshiwsky, M. I., & Packard, R. D. (1988, April). Predictive achievement model. Paper 
presented to the Career Ladder Steering Committee, Sunnyside School District, Tucson, 
Arizona. 

Packard, R. D. (1988, April). Research & evaluation: Pre paration nf th* network tasfr f.i T ff 
decision package on research content and pmrPd, l1T s , Document presented to the State 
Career Ladder Task Force Research Committee. 

Packard, R. D., & Dereshiwsky, M. I. (1988, April 30). Evaluation research: Study Of thfi 
effects Of a career ladder jntervention program with fivns o n tht» production ft nntmniw 
i n student ach i evement . Research document presented at the Arizona Educational 
Research Association for the Conference on Partnerships in Education, Tucson, Arizona. 

Karp,S. (1988, May). The effects of change on school climate: A res* sh,d y of a career ladder 
i nnovat i on . Unpublished doctoral dissertation, Northern Arizona University, Flagstaff, 
Arizona. (Cropper, Dissertation Chair.) 

Packard, R. D. (1988, Fall). "Career ladders: One form of educational reform." Excellence in 
leaching, 6 (No. i),4-7. 

Packard, R.D..& Dereshiwsky, M.I. (1988, October) Forces of rhanpe an d gnmifiEchaagg A 
Comprehensive school effectiveness propram f or enhanrgd teacher and st,.dp n t 
development - Document presented at the Arizona Education Association Leaders' 
Conference, Hyatt Regency, Phoenix, Arizona. 

Packard, R. D., & Dereshiwsky, M. L (1988, October) Program designs and strnr.t„n>c 
Document presented to the Joint Legislative Committee on Career Ladders, Arizona State 
Capitol, House Wing, Phoenix, Arizona. (ERIC Document Reproduction Service No. 
ED 302 534.) 

Packard, R. D„ Dereshiwsky, M. I., Gonzales, M., & Fimbres, E. (1988, November). EeseajcJi 
fi ndings On effective Program designs: Evaluating s tate and lnr.al program imparl o n 
profession a l development and improved student ontromoc p a pe r preserted to the 
National Council of States, New Orleans, Louisiana. (ERIC Document Reproduction 
Service No. ED 302 532.) 

Packard, R. D., Dereshiwsky, M. L, & Bas/Isaac, E. (1988, November) An integrated mmM fo r 
the Professional development of teacher leaders. Paper presented to the National Council 
of States, New Orleans, Louisiana. (ERIC Document Reproduction Service No. ED 302 



Packard, R. D., & Dereshiwsky, M. I. J1988, November 29). Quantitative levels of program 
acceptability by career ladder placement . Document presented to the Joint Legislative 
Committee on Career Ladders, Arizona State Capitol, House Wing, Phoenix, Arizona. 
(ERIC Document Reproduction Service No. ED 302 530.) 
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Packard, R. D., & Dereshiwsky, M. I. (198S, November 20) Quantitative te v el* <rf prog ram 
2g P3Ptafrflitv by compon ent s, and demographic conditions . Docur.ient presented to the 
Joint Legislative Committee oo Career ladders, Arizona State Capitol, House Wing. 
Phoenix, Arizona. (ERIC Docuc*6: Reproduction Service No. ED 302 535.) 

Packard, R.D. f &3Dereshiwsky,M.L (1988, November 29) A focused desipi to improve tgachs i 
dgvelOPTiynt and Stufcnt achievement for enhanced sc hool effectiveness Document 
presented to the Joint Legislative Committee on Career Ladders, Arizona State Capitol, 
House Wing, Phoenix, Arizona. (ERIC Docrment Reproduction Service No. ED 302 

Packard, R. D., & Gonzales, M. (1938, December 12). Planning models and procures fo r 
teacher inceptive and development programs A series of studies presented to the Crane 
School District, Yuma, Arizona. 

Kundin, K. R. (1988). The Arizona career ladder project: P erceptions of educators anr? 

2 *r , inistrators Year two of the pilot program - Unpublished doctoral dissertation, Arizona 
State University, Tempe, Arizona. 

Packard, R. D., & Dereshiwsky, M. I. (1988). A reliability assessment of the subscales of the 
1988 perception assessment scale survey. 

Packard, R. D., & Dereshiwsky, M. I. (W88). A factor analysis of the suhscales of the 1988 
perception assess ment scale survey. 

Packard, R. D., & Dereshiwsky, M. I. (1988). Demographic s ubgroup difference testing- 
Summated SCalft Scores derived from the 1988 pe rception assessment scale survey 



1989 - Advancement fn Prof ect Focus. Research 
Design and Methodology 

The Focus of Teacher Performance an d Student Achievem ent 

A leek of knowledge and application of adequate procedures related to measurement science and 
technology became apparent as a result of the presumed legislative intent to connect teacher performance to 
student achievement School districts across the country continue to make the assumption that when 
teachers are evaluated on their instructional processes, there is automatically a valid connection to student 
achievement More advanced research designs and models using multivariate statistical analysis procedures 
are being applied within three dissertations at the three state universities. These quantitative methods are 
more realistic, since ihey enable researchers to assess the joint effects of variables such as CL participation, 
years of teaching experience, and grade level taught In addition, multivariate methods allow for inclusion 
of multiple associated measures of student r^ormarice, such as a standardized test and a teacher-made test 
Preliminary results indicate a potential breakthrough in support of initial legislative opinions that teachers 
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should be evaluated and rewarded based on levels of performance, rather than solely on yean; of experience 
and college ciedit hours. 

Individualized A^e^p^t Through Case * Study Methodrrfoe y 

Data originally collected through the Perception Assessment Scale snrvgy identified the extreme 
diversities of districts with respect to readiness levels of the support and focus factors of effective change 
processes (see Figure 2 on page 19 for a depiction of the model). Additional research and data-analysis 
procedures were necessary in order to provide a more comprehensive picture of the actual program 
implementation processes operating within these individual districts. Through cose study methodology the 
researchers have piloted procedures which have provided valuable insights ; ^to the "opeiational functioning 
levels" of key support and focus components. Focus-group interviews conducted with all relevant sub- 
groups within a given system have provided an in-depth picture of participants' perceptions and feelings 
concerning their CL program experiences. Matrix analysis procedures have been used to summarize the 
open-ended responses to the Perception Assessment Scale survey, in order to identify the most commonly 
occurring areas of satisfaction and dissatisfaction with district- and school-level organizational climate. 
Such clustering and comparative analysis of qualitative (non-numeric) data represent the latest methodologic 
advances for understanding and reporting key participants 1 experiences in their own words. The result is a 
more valid, complete and rich data base, adding to existing knowledge in the social and behavioral sciences. 

Convergence and Profiling from Quant itative and Qualitative Dat a 

The foregoing variety of quantitative and qualitative data-analysis procedures, as applied to policy 
and evaluation research issues, represent the current state-of-the-art The results of applying such multiple 
and diverse methods as survey, case-study and model-validation techniques are continually assessed for 
consistency in order to determine the extent of methods convergence. A high degree of methods 
convergence means that the application of a number of different analytic techniques and data collection 
procedures has resulted in similar answers to the research question under study. This effectively indicates 
cross-validation, allowing for a corresponding high level of confidence in the conclusions. 
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An especially promising avenue of current assessment research involves the development of a 
profile for each district, based on specific quantitative and qualitative data resulting from the evaluation 
process. (Figures 3 and 4, pages 20 and 21, contain examples of such profiles.) Individual components 
(support and focus factors) of the readiness model are graphically depicted in terms of their cumulative 
frequency of mention as being positive or negative. That is, the position of each factor relative to the 
baseline of zero indicates its direction and magnitude of impact upon district readiness for change. By 
examining the components of these profdes, one can identify at a glance those support and focus factors 
which are operating at satisfactory levels, and whic are in need of remediation. The primary finding 
depicted in this profile is that those factors wliich should be supportive of teacher and student progress have 
instead turned out to have a negative imp? ; ~: 

A number of other research studi- are currendy underway; these are listed in the "research in 
progress" section. Topics being investigated include factors related to assessment of student achievement, 
measurement of teacher effectiveness, and identification of attitudes and opinions of educational policy 
leaders. 
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Research in Progress 

Planning Documentation in to the 1 OOP's 
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effective schools . Doctoral dissertation in progress, Northern Arizona University, 
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Northern Arizona University, flagstaff, Arizona. (Packard, Dissertation Chair.) 
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van a blgS and attJtUdinal Components of the 1988 perceptio n assessment gra1 f mf y A 
research document in progress. 

Packard, R. D., & Dereshiwsky, M. I. (1989/1990). Qualitative rssfiaffih hy r^ r 

distant. A research document in progress. 
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progress, Northern Arizona University, Flagstaff, Arizona. (Packard, Dissertation Chair.) 
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reform related to organizational support factors, Doctoral dissertation in progress, 
Northern Arizona University, Flagstaff, Arizona. (Packard, Dissertation Chair.) 



Summary and Conclusions 



A Potential B reakthrough 

Research and evaluation of the pilot-test program over a period of time has resulted in the 
following three major findings: 

1* Successful change and reform can be influenced by intervention programs 
such as career ladders, but it is primarily dependent upon well-developed 
and effective school systems. 

Development of effective schools includes the following key variables: a) competent and 
supportive leadership; b) clear and positive communications and interpersonal relationships; c) 
reliable and valid evaluation of teachers andklministrators; d) accurate application of advanced 
science and technology; e) curriculum and instruction whi';h meets local student needs; and f) 
reliable and valid accountability for student achievement 

2. The association between teacher performance and competency based on 
process and developmental evaluation can scientifically be related to 
reliable and valid student academic achievement measures. 

Many organizational bureaucracies, including school systems, compensate personnel 
based solely on years of experience and formal course work. They are not predominantly 
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rewarded based on competency and performance. Unfortunately, most districts are still 
operating on the assumption that mere observation of teachers 1 instructional processes which 
result in good teacher evaluations automatically imply a high rate of learning for their 
students. The preferred alternative, which has not been adopted to the extent wananted, is to 
examine the magnitude of student achievement gains. In most districts, more valid and 
reliable methods of evaluating teacher performance and student achievement are needed 
3. While the Career Ladders Pilot-Test Program does an excellent job of 
assuring teacher accountability, it has been far less successful with respect 

w 

to assessing the corresponding accountability of other key factors of 
district operational effectiveness. 

Teachers have effectively been held accountable in terms of time on task, expanded 
responsibilities focused on instructional improvement, good classroom teaching and 
communication skills and demonstrating student achievement gains. However, the program 
has not held districts accountable for the development of an adequate curriculum, for 
improvement in communications and governance operations (participatory management, 
shared responsibilities, etc.) or for the development of long-range plans for demonstrating 
student achievement gains in individual schools as well as across the total district More 
effective indicators of district-wide progress would enhance the evaluative aspects of the 
program. 

Review ofPmpess 

From 1985 to the present, the career ladders pilot-test research and evaluation methodology has 
evolved from a general evaluation design, to identification and description of relevant concepts and processes 
and in-depth into a variety of data collection, analysis and reporting procedures. Since career ladders 
program success was found to be dependent primarily upon factors other than the legislative and district 
goals and district CL plans and structures, current research has centered on essential support and focus 
factors which must be operating effectively before a ' acher incentive and development program can 
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progress. It has become apparent that an effective school organization and operation is necessary in order 
for all participating districts to be equally successful in carrying out the legislative intent of the CL policy. 

Epilogue 

Democracy and the Freedom to 
Pursue Individualities 
and Uniquenesses 

Politically, in a democracy, individuals and groups, 
with their uniquenesses and diversities, are equally 
recognized in the pursuit of freedom and independence. 

Socially and behaviorally, in a democracy, these 
diversities must be recognized and allowed the 
freedom to develop in the direction of their 
uniquenesses and at the rate of their own potentialities 
for growth. 

Educationally, equal treatments or interventions 
placed on unequal (unique) individuals, socially and 
v ^haviorally, fail to recognize diversities and 
uniquenesses and the way in which human beings and 
oiganizations develop. 

To make everyone developmentally equal would 
require genetic and environmental interventions and 
manipulation by powerful non-democratic forces; by 
doing so, everyone would have an externally mandated 
potential to become socially and behaviorally "equal." 

Equal treatment in intervention and developmental 
time, applied to unequal and unique entities, does not 
democratically, socially or behaviorally recognize the 
uniqueness and diversity of individuals or groups 
within our free society. 

There is no way to recover from being socially and 
emotionally unequal, and in a democracy it is 
definitely not necessary. All individuals must have 
the freedom, independence and support to develop 
their own potential. 

Richard D. Packard, 1989 




Dr. Richard D. Packard, 1988 
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